Systems and Methods for Version Chain Clustering

ABSTRACT

A system, a method and a computer program product for storing data, which include receiving a data stream having a plurality of transactions that include at least one portion of data, determining whether at least one portion of data within at least one transaction is substantially similar to at least another portion of data within at least one transaction, clustering together at least one portion of data and at least another portion of data within at least one transaction, selecting one of at least one portion of data and at least another portion of data as a representative of at least one portion of data and at least another portion of data in the received data stream, and storing each representative of a portion of data from each transaction in the plurality of transactions, wherein a plurality of representatives is configured to form a chain representing the received data stream.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Patent ProvisionalApplication No. 61/534,166 to Ramu et al., filed Sep. 13, 2011, andentitled “Version Chain Clustering” and incorporates disclosure of thisapplication herein by reference in its entirety.

TECHNICAL FIELD

The subject matter described herein relates to data processing and inparticular, to more efficiently process data contained indelta-compressed version chains through version chain clustering.

BACKGROUND

Data processing applications allow their users to create, change,modify, and delete files over time. A file version represents aparticular iteration of a file at a point in time. Such iterations canbe the same or can be different from the originally created file and/orfrom its other versions. Some files may have no versions (i.e., only asingle original file), only a few versions, or a plurality of versions.An efficient way to store versions of files or segments of files overtime is by delta compressing versions against each other and storingthem in a version chain. Version chains are typically linear datastructures that hold contents of versions of the same or similar filesor segments of files over time. For example, a segment that isoriginally created and then modified four times can have a version chainconsisting of four versions, which would represent a version of the fileor file segment at four different points in time.

To reduce storage space, file versions are typically stored in acompressed format, such as a delta-compressed format. Delta compressionor delta encoding is a way of storing or transmitting data in the formof differences between versions of a file or file segment rather thancomplete files. The differences are recorded in discrete files, whichare called “deltas.”

In some cases, the version chains can be represented as linear reversedelta version chains, where the most recent version is stored in itswhole form and all earlier versions could be stored as difference/deltafiles from each other in the linear chain. Conversely, a forward deltaversion chain maintains the first version of a file in its whole form,and creates delta files forward from that first version.

While a linear arrangement of delta versions can be one of the simplestdata structures for version chains, there are operations on versionchains that make the linear arrangement of deltas inefficient, moreprone to data loss, and/or cumbersome, as indicated below.

One of these operations includes accessing an earlier version of a file,which is a linear process whose processing time is directly proportionalto the position of that version along the length of the version chain.The shorter the distance from the most recent version to the desiredversion within the reverse delta version chain, the faster the executiontime to recreate that earlier version. However, this operation canconsume a greater amount of time and processing power as the distancefrom the most recent version to the desired version increases.

Another operation includes deleting a single delta version from anywherein the version chain except the ends of the chain. This can requiredecompressing of all more recent versions of the version to be deletedin order to remove that version and reconnect its two adjacent versionsto each other. This can again be a time-and-processing intensiveoperation.

If a delta version within a version chain is determined to have becomecorrupted, all earlier versions are rendered unavailable since theirregeneration is based on all of the more recent versions to be errorfree. Hence, there is a need to reduce the probability of data loss bysignificantly reducing the number of deltas that must be error free inorder to successfully restore an earlier version of a segment or file.

Version chains are typically arranged in a linear format. Version chainscan also be implemented in a binary tree data structure to reduce theoverall time in accessing earlier versions. However, if a primary goalof a version chain is to minimize data storage capacity, and it isassumed that two versions adjacent in time can produce a smaller deltafile than two versions separated by a larger period of time, then abinary tree version chain can produce suboptimal data storagecompression.

Thus, there is a need for a system and method for storing data thatinvolves an improved delta version chain data structure, where thestructure can be configured to mitigate various issues with the linearand binary-tree structures discussed above.

SUMMARY

In some implementations, the current subject matter relates to acomputer-implemented method for storing data. The method includesreceiving a data stream, wherein the data stream includes a plurality oftransactions, each transaction in the plurality of transactions includesat least one portion of data from the received data stream, determiningwhether the at least one portion of data within at least one transactionin the plurality of transactions is substantially similar to at leastanother portion of data within the at least one transaction, clusteringtogether the at least one portion of data and at least another portionof data within the at least one transaction, selecting one of the atleast one portion of data and the at least another portion of data asbeing a representative of the at least one portion of data and the atleast another portion of data in the received data stream, and storingeach representative of a portion of data from each transaction in theplurality of transactions, wherein a plurality of representatives isconfigured to form a version chain representing parts of the receiveddata stream. At least one of the receiving, the determining, theclustering, the selecting, and the storing is performed on at least oneprocessor.

In some implementations, the current subject matter can be configured toinclude at least one of the following optional features. At least oneportion of data can be at least one compressed portion of data and atleast another portion of data can be at least another compressed portionof data. The representative can be a compressed representative of the atleast one portion of data and the at least another portion of data. Therepresentative can be a delta-compressed representative of at least oneportion of data and at least another portion of data. The portions ofdata that are not selected representatives can be configured to beindependent of each other, thereby reducing a number of dependenciesamong the portions of data that are not selected representatives.

The method can include determining whether a third portion of dataincluded in at least one transaction within the plurality of transactionis designated for deletion, determining a representative that designatesthe third portion of data, and deleting the representative thatdesignates the third portion of data, and deleting all portions of datathat the representative that designates the third portion of data isconfigured to represent. The method can include determining whether athird portion of data included in at least one transaction within theplurality of transaction is designated for deletion, determining arepresentative that designates the third portion of data, and deletingthe third portion of data without deleting the representative thatdesignates the third portion of data or other portions of data that therepresentative that designates the third portion of data is configuredto represent. At least one portion of data can be configured to be aversion of a data file within the received data stream and at leastanother portion of data can be configured to be another version of thedata file within the received data stream. The version of the data fileand another version of the data file can be compressed.

The storing can further include storing compressed versions of the datafile in at least one storage location.

The method can include uncompressing the representative of the at leastone portion of data and the at least another portion of data,retrieving, based on the uncompressing, at least one of an uncompressedversion and an uncompressed another version of the data file from the atleast one storage location. The processing time for the retrieving canbe based on a number of representatives configured to represent at leastone transaction in the plurality of transactions and configured to beuncompressed to retrieve the uncompressed version.

The method can further include repeating the determining, theclustering, and the selecting for each portion of the data in thereceived data stream.

The method can also include storing portions of data that are notselected as a representative.

In some embodiments, the clustered at least one portion of data and theclustered at least another portion of data are configured to have nodependencies on one another.

Articles are also described that comprise a tangibly embodiedmachine-readable medium embodying instructions that, when performed,cause one or more machines (e.g., computers, etc.) to result inoperations described herein. Similarly, computer systems are alsodescribed that can include a processor and a memory coupled to theprocessor. The memory can include one or more programs that cause theprocessor to perform one or more of the operations described herein.

The details of one or more variations of the subject matter describedherein are set forth in the accompanying drawings and the descriptionbelow. Other features and advantages of the subject matter describedherein will be apparent from the description and drawings, and from theclaims.

Articles are also described that comprise a tangibly embodiedmachine-readable medium embodying instructions that, when performed,cause one or more machines (e.g., computers, etc.) to result inoperations described herein. Similarly, computer systems are alsodescribed that can include a processor and a memory coupled to theprocessor. The memory can include one or more programs that cause theprocessor to perform one or more of the operations described herein.

The details of one or more variations of the subject matter describedherein are set forth in the accompanying drawings and the descriptionbelow. Other features and advantages of the subject matter describedherein will be apparent from the description and drawings, and from theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, show certain aspects of the subject matterdisclosed herein and, together with the description, help explain someof the principles associated with the disclosed implementations. In thedrawings,

FIG. 1 illustrates an exemplary linear version chain.

FIG. 2 illustrates an exemplary linear version deletion from a versionchain operation.

FIG. 3 a illustrates an exemplary linear version chain, according tosome implementations of the current subject matter.

FIG. 3 b illustrates an exemplar clustered version chain based on theversion chain shown in FIG. 3 a, according to some implementations ofthe current subject matter.

FIG. 4 illustrates an exemplary system for storing data, according tosome implementations of the current subject matter.

FIG. 5 illustrates an exemplary method for storing data, according tosome implementations of the current subject matter.

DETAILED DESCRIPTION

To address these and potentially other deficiencies of currentlyavailable solutions, one or more implementations of the current subjectmatter provide methods, systems, articles or manufacture, and the likethat can, among other possible advantages, provide systems and methodsfor providing systems, methods, and computer program products forstoring data.

There are many applications that leverage version chains and deltacompression. Software source control systems efficiently manage multipleversions of source files over time so that a user is capable ofacquiring any earlier version of a file. Backup systems can store day today backups in version chains using delta compression to eliminateredundancy among successive backups. Computer file systems have alsobeen designed to maintain the history of files over time using versionchains. The current subject matter can be configured to work withreverse delta version chains and/or forward delta version chains.

For any of these applications, within the stream of data to beversioned, there can exist segments that can be similar or substantiallyidentical in content. By identifying these similar/identical segmentsand grouping each of them together into version chains, and applyingdelta compression between successive pairs of historical orcontent-similar versions, the overall consumed capacity of these similarsegments can be dramatically reduced by factors up to 1,000,000:1 orgreater.

Each stream of data can be segmented into smaller logical entities,which can be referred to as transactions. For example, a weekend backupof 20 terabytes (“TB”) of data may be segmented into 200 transactions of100 gigabytes (“GB”) each. Transactions can be entities within the datastream that can be created and/or deleted/purged as a complete unit.Distinct segments of data within each transaction can be similar orsubstantially identical to one another.

Conventional systems typically serially link a transaction's segments ofidentical or similar data into a version chain as well as segments thatare found to be similar or identical among multiple transactions. Thishas the potential to create version chains that are thousands ofversions or more in length. The length of a version chain can beproblematic for operations such as deleting a version from the middle ofa version chain or recreating an earlier version of a file. Also, in achain of a thousand versions, if a single version is found to havebecome corrupt in the linear list of versions, all earlier versions canbe rendered unavailable.

In some implementations, to address the above, the current subjectmatter can be configured to recognize that the data within eachtransaction of a data stream can have similar or substantially identicalcontent and can use that similarity within a transaction to create acluster of versions with a single version being attached to a main trunk(which can include a collection of versions of data) of the serialversion chain. One of the advantages of such clustering can include areduction of an amount of processing involved to delete one or moreversions from a version chain, where the reduction depends on a degreeof clustering. Another advantage is the reduction in processing time forrestoring an early version of a file within the chain, since fewer deltadecompressions have to be performed. A further advantage of the currentsubject matter can include reducing a probability of data loss bysignificantly reducing the number of deltas that must be error free inorder to restore an earlier version. A shorter serial main trunk canreduce a probability of a single version corrupting all other earlierversions by the factor of the degree of clustering. In someimplementations, the current subject matter relates to a method, asystem and a computer program product for storing data within a versionchain that can minimize the length of the version chain and allow a moreefficient processing of the versions within a version chain.

FIG. 1 illustrates an exemplary conventional linear version chain 100configured to employ delta compression. The version chain 100 includes amain trunk 104 having a most recent version 101 (represented by a squarein FIG. 1) and a plurality of earlier versions 102 (represented bytriangles). The version chain 100 can represent an efficient storage often versions of a single file. It can also represent ten versions ofsegments of data from a data stream that can be similar to one another.The most recent version 101 of a data file can be stored in its wholeform (e.g., in its entirety) and/or can be data compressed using acommon data compression algorithm, such as, gzip, Lempel-Ziv, and/or anyother suitable method. The plurality of earlier versions 102 can bestored as difference and/or delta compressed files that are created byonly storing the different bytes between successive versions.

To recreate an entire previous version 103, the most recent version 101and the delta (or stored difference) from the previous version 103 canbe combined and the combination can be un-delta-compressed to turn thatdelta into a full file. Thus, recreation of the first version 105 of thefile, can involve similar combination and decompression operations ofall versions 102 of the file in the serial link of versions between file101 and the version 105. As shown in FIG. 1, nine different combinationand delta-decompression operations can recreate the original file.Larger version chains can involve greater number ofcombination/decompression operations and thus, can consume greateramount of processing time and power. One of the advantages of thecurrent subject matter is that it can reduce the serial length of aversion chain through version chain clustering and can accelerate there-creation of earlier versions within a version chain.

FIG. 2 illustrates an exemplary conventional linear version chain wherea delete operation of a version within a linear version chain must beperformed. The version chain 200 can represent a collection of tenversions of a file or file segment over time. FIG. 2 illustrates threephases 201, 203, and 204 during which a version 202 (represented bytriangle bearing “6”) is deleted from the version chain 200. In phase201, a version 202 (triangle 6) is selected from the version chain 200for deletion. Similar to FIG. 1, the most recent version is representedby the square bearing number “10” and its previous versions arepresented by triangles bearing numbers 1-9, with triangle 1 being theearliest version of the file and triangle 9 being the version that isjust prior the file's version 10. Version 10 can be a full version ofthe file and/or compressed version. Versions 1-9 can be represented bydeltas or differences between the successive versions. A version of afile can be deleted if, for example, a determination is made that suchversion is no longer necessary to maintain. Hence, this version can bedeleted to conserve storage capacity.

In phase 203 of the version deletion process shown in FIG. 2, toeliminate version 6 in the version chain 200, all versions from version10 to version 5 need to be serially delta-decompressed working fromversion 10 back to version 5, as shown by the squares 5-10, whichindicate that the deltas represented by triangles 5-10 have beendecompressed into full versions. Once versions 5-9 have beendecompressed, version 6 can be deleted and delta-compression can beapplied to the remaining versions 5 and 7-9 to create a new shorterversion chain, as shown in phase 204. FIG. 2 further illustrates thatlonger version chains with thousands of versions can consume an amountof time to process a delete operation on an earlier version that isproportional to the position of version to be deleted within the versionchain. Thus, shorter version chains can make for more efficient versiondeletion operations.

FIG. 3 a illustrates an exemplary version chain 301 prior to clustering,according to some implementations of the current subject matter. Thesimilar files and file segments within and among multiple transactionsof one or more data streams have been processed into a version chain301. An exemplary version chain 301 is shown in FIG. 3 a. The versionchain 301 includes a most recent version 318 (shown as a square in FIG.3 a) of the data file and a plurality of deltas 311, 313, and 315 (shownas triangles in FIG. 3 a). The most recent version 318 and plurality ofdeltas 311, 313, and 315 are maintained in a linear configuration ofversions. As shown in FIG. 3 a, the version chain 301 includes 16versions of a data file, including the most recent version 318. Theearliest version of the file (i.e., the earliest point in time) isrepresented by the triangle 319 and is disposed at the end of theversion chain 301. Conventional version chain storage systems storeversion chain in a linear fashion, whereby to re-create the original,first version in the version chain, n−1 combinations anddelta-decompressions need to be performed, where n is a number ofversions in the version chain.

The versions in the version chain 301 can be grouped by theirco-residence within a transaction of the original data stream. Eachversion chain can be reorganized to include fewer delta files along amain trunk of the version chain. As shown in FIG. 3 a, within versionchain 301, three separate transactions are being maintained (transactionA 312, transaction B 314, and transaction C 316). By grouping all of thedeltas representing files or file segments within each transaction intotheir own version cluster, and attaching any one of these deltas withineach cluster to the main trunk 310 of a version chain, the effectivelength of the version chain can be dramatically reduced. For example,the four delta versions 313 of transaction B in FIG. 3 a can be arrangedwith one delta version 303 b of the four deltas 314 residing on the maintrunk 310 of the version chain in FIG. 3 b. The remaining three deltaswould be computed as the difference bytes between their content and file303 b.

FIG. 3 b illustrates an exemplary version chain 301 after clustering hasbeen applied, according to some implementations of the current subjectmatter. In some implementations, clustering can include selection ordesignation or identification of a single version within a transactionto represent all versions in that transaction. Any version in themultiple versions in a transaction can be selected as a representativeof all versions in that transaction. Hence, the selection or designationor identification of which of multiple versions becomes therepresentative on the main trunk of the clustered version chain is notcritical, and need not even be consistent. In some implementations, themost recent version or the first version can be chosen for convenience.For example, version 303 a can be present on the main trunk of theversion chain. All versions 311 can be computed as content differencesbetween themselves and version 303 a in the transaction A 312, therebyforming cluster 304 a. Likewise, version 303 b can be present on themain trunk of the version chain. All versions 304 b would be computed ascontent differences between themselves and version 303 b in thetransaction. Likewise, version 303 c can be present on the main trunk ofthe version chain. All versions 304 c that are not a representativeversion can be computed as content differences between themselves andversion 303 c in the transaction. The versions within each cluster thatare not representative versions are configured to be independent of oneanother and thus, retrieval, deletion, modification, and/or any otheroperation on a single version within the cluster does not affect anyother versions in that cluster or other clusters within the clusteredversion chain.

A clustered version chain can greatly reduce the time it takes torecreate earlier versions of files. To recreate the original version 319using linear version chain shown in FIG. 3 a, it can take 15 deltadecompression operations. To recreate any of the versions 311 (which caninclude the original version 319) using some embodiments of the currentsubject matter shown in FIG. 3 b, it can take 3 delta decompressionoperations. Thus, this yields an effective speedup factor of 5.

A clustered version chain can also greatly reduce the time it takes todelete various delta versions within a version chain. Within a singlecluster, any of the leaf node deltas (304 a, 304 b, 304 c) that are tobe deleted can simply be deleted without the need to delta decompressany other files.

Another common version deletion request can include elimination of alldelta files associated with a specific transaction. Instead of having toserially delta-decompress and eliminate each version within atransaction in the linear version chain, with the clustered versionchain, just the representative deltas that are on the main trunk 310 canbe involved in the deletion of the version that is on the main trunkthat represents that transaction.

In some implementations, to retrieve a particular version of a data filefrom a version chain, the current subject matter's system can beconfigured to determine its designated representative. For example, toretrieve a version of a file in the cluster 304 a, the current subjectmatter can determine that its designated representative version is 303a, as shown in FIG. 3 b. Using the designated representative 303 a, thecurrent subject matter's system can be configured to retrieve a versionfrom the cluster represented by the representative 303 a. This greatlyincreases speed and efficiency of the retrieval process, as the currentsubject matter's system decompresses versions that are stored on themain trunk 310, i.e., designated representative versions 303 a, 303 b,303 c (if version 303 c is a full uncompressed version of the data file,it need not be decompressed) as well as delta-decompresses versions thatcorrespond to the desired version within the cluster. For example, ifversion 311 a in the cluster 311 is desired to be retrieved, the currentsubject matter's system can be configured to decompress version 303 b,then decompress version 303 a and combine it with decompressed version303 b, and then decompress version 311 a and combine it with thedecompressed versions 303 a and 303 b. Thus, acquiring an earlierversion of a file using the current subject matter's system is moreefficient based on the degree of clustering. For example, retrieving thefirst version of a file (version 319 shown in FIG. 3 a or version 311 ashown in FIG. 3 b), the number of delta compression operations isreduced from fifteen in the linear version chain (FIG. 3 a) to threeoperations. This is significant savings of the processing time andsystem power.

In some implementations, deletion of a particular version can also beaccelerated using the current subject matter's system. Deletions formany applications can be transactional in nature. This can be true ofbackup data sets. Deletion of an entire transaction can be minimized bythe shorter length of the main trunk. Further, by deleting the node of atransaction on the main trunk, all other versions can be quickly deletedsince they only depend on the version attached to the main trunk.

In the linear version chain 301 shown in FIG. 3 a, deletion of allversions from transaction B 314 can involve ten delta de-compressionoperations, 4 deletions and a delta compression operation to connect themost recent version of transaction A 312 to the least recent version oftransaction C 312. In the clustered version chain 301 shown in FIG. 3 b,the deletion of transaction B can involve two delta-decompressoperations, with the same number of deletion and delta compressionoperations to attach the main trunk version of transaction A 312 to themain trunk version of transaction C 316. Deletions of a particularversion within a cluster of versions can also be made using the versionscontained on the main trunk. For example, version 313 a can simply bedeleted since it has no dependencies. Further, all versions 304 a can bedeleted without any effect on versions 304 b or 304 c, as versions 304a, 304 b, and 304 c do not depend on one another.

In addition to version chain operation efficiency, the clustered versionchain can reduce the probability of version chain data loss when asingle version becomes corrupt anywhere in the chain. Since therecreation of any earlier version in the chain can require that all morerecent versions be substantially error free, a linear version chain isless reliable than the clustered equivalent. The first version 319 inthe linear version chain 301 shown in FIG. 3 a can require that all ofthe versions in the version chain be error free in order for it to berecreated in its whole form. In the clustered version chain 301 shown inFIG. 3 b, only the versions that are on the main trunk 310, i.e.versions 303 a, 303 b, 303 c (if version 303 c is not an uncompressedversion) and the single version in a selected cluster 304 a, 304 b, 304c of versions that needs to be recreated must be substantially errorfree. Thus, the linear version chain 301 shown in FIG. 3 a can requirethat all sixteen versions be error free while in the clustered versionchain 301 shown in FIG. 3 b, at most only four versions can be requiredto be substantially error free.

In some implementations, the current subject matter can be configured tobe implemented in a system 400, as shown in FIG. 4. The system 400 caninclude a processor 410, a memory 420, a storage device 430, and aninput/output device 440. Each of the components 410, 420, 430 and 440can be interconnected using a connection 450. The processor 410 can beconfigured to process instructions for execution within the system 400.The processor 410 can be further configured to process variousinstructions and perform operations, including those shown in FIG. 5, aswell as those that are stored in the memory 420 or on the storage device430, including receiving or sending information through the input/outputdevice 440. The memory 420 can store information within the system 400.In some implementations, the memory 420 can be a computer-readablemedium, a volatile memory unit, a non-volatile memory unit, or any othersuitable medium/unit. The storage device 430 can be capable of providingmass storage for the system 400 and can be a computer-readable medium, afloppy disk device, a hard disk device, an optical disk device, a tapedevice, non-volatile solid state memory, or any other suitable storagedevice. The input/output device 440 can be a network interface and/orany other system, device, and/or any combination thereof.

In some implementations, the current subject matter relates to acomputer-implemented method 500 for storing data. At 502, a data streamcan be received, wherein the data stream includes a plurality oftransactions, each transaction in the plurality of transactions includesat least one portion of data from the received data stream. At 504, adetermination can be made whether the at least one portion of datawithin at least one transaction in the plurality of transactions issubstantially similar to at least another portion of data (or a anotherversion of data or another delta-compressed version of data or anotherdelta/difference representing another version of data) within the atleast one transaction. At 506, at least one portion of data and at leastanother portion of data within the at least one transaction can beclustered together. At 508, one of the at least one portion of data andthe at least another portion of data can be selected as being arepresentative (e.g., representative 303 a, 303 b, or 303 c shown inFIG. 3 b) of the at least one portion of data and the at least anotherportion of data in the received data stream. At 510, each representativeof a portion of data from each transaction in the plurality oftransactions can be stored. A plurality of representatives can beconfigured to form a chain representing the received data stream. Atleast one of the receiving, the determining, the clustering, theselecting, and the storing is performed on at least one processor.

In some implementations, the current subject matter can be configured toinclude at least one of the following optional features. At least oneportion of data can be at least one compressed portion of data and theat least another portion of data is at least another compressed portionof data. The representative can be a compressed representative of the atleast one portion of data and the at least another portion of data. Therepresentative can be a delta-compressed representative of at least oneportion of data and at least another portion of data. The portions ofdata that are not selected representatives can be configured to beindependent of each other, thereby reducing a number of dependenciesamong the portions of data that are not selected representatives (insome embodiments, all dependencies among the portions of data that arenot selected representatives can be eliminated).

The method can include determining whether a third portion of dataincluded in at least one transaction within the plurality of transactionis designated for deletion, determining a representative that designatesthe third portion of data, and deleting the representative thatdesignates the third portion of data, and deleting all portions of datathat the representative that designates the third portion of data isconfigured to represent. The method can include determining whether athird portion of data included in at least one transaction within theplurality of transaction is designated for deletion, determining arepresentative that designates the third portion of data, and deletingthe third portion of data without deleting the representative thatdesignates the third portion of data and other portions of data that therepresentative that designates the third portion of data is configuredto represent. At least one portion of data can be configured to be aversion of a data file within the received data stream and at leastanother portion of data can be configured to be another version of thedata file within the received data stream. The version of the data fileand another version of the data file can be compressed. The storing canfurther include storing compressed versions of the data file in at leastone storage location (e.g., a repository, a memory, a disk, a physicalstorage, a virtual storage, and/or any other storage system, device,and/or mechanism).

The method can include uncompressing the representative of the at leastone portion of data and the at least another portion of data,retrieving, based on the uncompressing, at least one of an uncompressedversion and an uncompressed another version of the data file from the atleast one storage location. The processing time for the retrieving canbe based on a number of representatives configured to represent at leastone transaction in the plurality of transactions and configured to beuncompressed to retrieve the uncompressed version.

The method can further include repeating the determining, theclustering, and the selecting for each portion of the data in thereceived data stream. The method can also include storing portions ofdata that are not selected as a representative.

In some embodiments, the clustered at least one portion of data and theclustered at least another portion of data are configured to have nodependencies on one another.

The systems and methods disclosed herein can be embodied in variousforms including, for example, a data processor, such as a computer thatalso includes a database, digital electronic circuitry, firmware,software, or in combinations of them. Moreover, the above-noted featuresand other aspects and principles of the present disclosedimplementations can be implemented in various environments. Suchenvironments and related applications can be specially constructed forperforming the various processes and operations according to thedisclosed implementations or they can include a general-purpose computeror computing platform selectively activated or reconfigured by code toprovide the necessary functionality. The processes disclosed herein arenot inherently related to any particular computer, network,architecture, environment, or other apparatus, and can be implemented bya suitable combination of hardware, software, and/or firmware. Forexample, various general-purpose machines can be used with programswritten in accordance with teachings of the disclosed implementations,or it can be more convenient to construct a specialized apparatus orsystem to perform the required methods and techniques.

The systems and methods disclosed herein can be implemented as acomputer program product, i.e., a computer program tangibly embodied inan information carrier, e.g., in a machine readable storage device or ina propagated signal, for execution by, or to control the operation of,data processing apparatus, e.g., a programmable processor, a computer,or multiple computers. A computer program can be written in any form ofprogramming language, including compiled or interpreted languages, andit can be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment. A computer program can be deployed to be executedon one computer or on multiple computers at one site or distributedacross multiple sites and interconnected by a communication network.

As used herein, the term “user” can refer to any entity including aperson or a computer.

Although ordinal numbers such as first, second, and the like can, insome situations, relate to an order; as used in this document ordinalnumbers do not necessarily imply an order. For example, ordinal numberscan be merely used to distinguish one item from another. For example, todistinguish a first event from a second event, but need not imply anychronological ordering or a fixed reference system (such that a firstevent in one paragraph of the description can be different from a firstevent in another paragraph of the description).

The foregoing description is intended to illustrate but not to limit thescope of the invention, which is defined by the scope of the appendedclaims. Other implementations are within the scope of the followingclaims.

These computer programs, which can also be referred to programs,software, software applications, applications, components, or code,include machine instructions for a programmable processor, and can beimplemented in a high-level procedural and/or object-orientedprogramming language, and/or in assembly/machine language. As usedherein, the term “machine-readable medium” refers to any computerprogram product, apparatus and/or device, such as for example magneticdiscs, optical disks, memory, and Programmable Logic Devices (PLDs),used to provide machine instructions and/or data to a programmableprocessor, including a machine-readable medium that receives machineinstructions as a machine-readable signal. The term “machine-readablesignal” refers to any signal used to provide machine instructions and/ordata to a programmable processor. The machine-readable medium can storesuch machine instructions non-transitorily, such as for example as woulda non-transient solid state memory or a magnetic hard drive or anyequivalent storage medium. The machine-readable medium can alternativelyor additionally store such machine instructions in a transient manner,such as for example as would a processor cache or other random accessmemory associated with one or more physical processor cores.

To provide for interaction with a user, the subject matter describedherein can be implemented on a computer having a display device, such asfor example a cathode ray tube (CRT) or a liquid crystal display (LCD)monitor for displaying information to the user and a keyboard and apointing device, such as for example a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well. For example,feedback provided to the user can be any form of sensory feedback, suchas for example visual feedback, auditory feedback, or tactile feedback;and input from the user can be received in any form, including, but notlimited to, acoustic, speech, or tactile input.

The subject matter described herein can be implemented in a computingsystem that includes a back-end component, such as for example one ormore data servers, or that includes a middleware component, such as forexample one or more application servers, or that includes a front-endcomponent, such as for example one or more client computers having agraphical user interface or a Web browser through which a user caninteract with an implementation of the subject matter described herein,or any combination of such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, such as for example acommunication network. Examples of communication networks include, butare not limited to, a local area network (“LAN”), a wide area network(“WAN”), and the Internet.

The computing system can include clients and servers. A client andserver are generally, but not exclusively, remote from each other andtypically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on therespective computers and having a client-server relationship to eachother.

The implementations set forth in the foregoing description do notrepresent all implementations consistent with the subject matterdescribed herein. Instead, they are merely some examples consistent withaspects related to the described subject matter. Although a fewvariations have been described in detail above, other modifications oradditions are possible. In particular, further features and/orvariations can be provided in addition to those set forth herein. Forexample, the implementations described above can be directed to variouscombinations and sub-combinations of the disclosed features and/orcombinations and sub-combinations of several further features disclosedabove. In addition, the logic flows depicted in the accompanying figuresand/or described herein do not necessarily require the particular ordershown, or sequential order, to achieve desirable results. Otherimplementations can be within the scope of the following claims.

1. A computer-implemented method for storing data, comprising: receivinga data stream, wherein the data stream includes a plurality oftransactions, each transaction in the plurality of transactions includesat least one portion of data from the received data stream; determiningwhether the at least one portion of data within at least one transactionin the plurality of transactions is substantially similar to at leastanother portion of data within the at least one transaction; clusteringtogether the at least one portion of data and at least another portionof data within the at least one transaction; selecting one of the atleast one portion of data and the at least another portion of data asbeing a representative of the at least one portion of data and the atleast another portion of data in the received data stream; and storingeach representative of a portion of data from each transaction in theplurality of transactions, wherein a plurality of representatives isconfigured to form a chain representing the received data stream;wherein the at least one of the receiving, the determining, theclustering, the selecting, and the storing is performed on at least oneprocessor.
 2. The method according to claim 1, wherein the at least oneportion of data is at least one compressed portion of data and the atleast another portion of data is at least another compressed portion ofdata.
 3. The method according to claim 1, wherein the representative isa compressed representative of the at least one portion of data and theat least another portion of data.
 4. The method according to claim 1,wherein the representative is a delta-compressed representative of theat least one portion of data and the at least another portion of data.5. The method according to claim 1, wherein the portions of data thatare not selected representatives are configured to be independent ofeach other, thereby reducing a number of dependencies among the portionsof data that are not selected representatives.
 6. The method accordingto claim 1, further comprising determining whether a third portion ofdata included in at least one transaction within the plurality oftransaction is designated for deletion; determining a representativethat designates the third portion of data; and deleting therepresentative that designates the third portion of data, and deletingall portions of data that the representative that designates the thirdportion of data is configured to represent.
 7. The method according toclaim 1, further comprising determining whether a third portion of dataincluded in at least one transaction within the plurality of transactionis designated for deletion; determining a representative that designatesthe third portion of data; and deleting the third portion of datawithout deleting the representative that designates the third portion ofdata and other portions of data that the representative that designatesthe third portion is configured to represent.
 8. The method according toclaim 1, wherein the at least one portion of data is configured to be aversion of a data file within the received data stream; and the at leastanother portion of data is configured to be another version of the datafile within the received data stream; wherein the version of the datafile and the another version of the data file are compressed.
 9. Themethod according to claim 8, wherein the storing further comprisesstoring compressed versions of the data file in at least one storagelocation.
 10. The method according to claim 9, further comprisinguncompressing the representative of the at least one portion of data andthe at least another portion of data; retrieving, based on theuncompressing, at least one of an uncompressed version and anuncompressed another version of the data file from the at least onestorage location.
 11. The method according to claim 10, wherein aprocessing time for the retrieving is based on a number ofrepresentatives configured to represent at least one transaction in theplurality of transactions and configured to be uncompressed to retrievethe uncompressed version.
 12. The method according to claim 1, furthercomprising repeating the determining, the clustering, and the selectingfor each portion of the data in the received data stream.
 13. The methodaccording to claim 1, further comprising storing portions of data thatare not selected as a representative.
 14. The method according to claim1, wherein the clustered at least one portion of data and the clusteredat least another portion of data are configured to have no dependencieson one another.
 15. A system for storing data, comprising: at least oneprocessor; and at least one machine-readable medium storing instructionsthat, when executed by the at least one processor, cause the at leastone processor to perform operations comprising: receiving a data stream,wherein the data stream includes a plurality of transactions, eachtransaction in the plurality of transactions includes at least oneportion of data from the received data stream; determining whether theat least one portion of data within at least one transaction in theplurality of transactions is substantially similar to at least anotherportion of data within the at least one transaction; clustering togetherthe at least one portion of data and at least another portion of datawithin the at least one transaction; selecting one of the at least oneportion of data and the at least another portion of data as being arepresentative of the at least one portion of data and the at leastanother portion of data in the received data stream; and storing eachrepresentative of a portion of data from each transaction in theplurality of transactions, wherein a plurality of representatives isconfigured to form a chain representing the received data stream. 16.The system according to claim 15, wherein the at least one portion ofdata is at least one compressed portion of data and the at least anotherportion of data is at least another compressed portion of data.
 17. Thesystem according to claim 15, wherein the representative is a compressedrepresentative of the at least one portion of data and the at leastanother portion of data.
 18. The system according to claim 15, whereinthe representative is a delta-compressed representative of the at leastone portion of data and the at least another portion of data.
 19. Thesystem according to claim 15, wherein the portions of data that are notselected representatives are configured to be independent of each other,thereby reducing a number of dependencies among the portions of datathat are not selected representatives.
 20. The system according to claim15, wherein instructions further comprise determining whether a thirdportion of data included in at least one transaction within theplurality of transaction is designated for deletion; determining arepresentative that designates the third portion of data; and deletingthe representative that designates the third portion of data, anddeleting all portions of data that the representative that designatesthe third portion of data is configured to represent.
 21. The systemaccording to claim 15, wherein instructions further comprise determiningwhether a third portion of data included in at least one transactionwithin the plurality of transaction is designated for deletion;determining a representative that designates the third portion of data;and deleting the third portion of data without deleting therepresentative that designates the third portion of data and otherportions of data that the representative that designates the thirdportion is configured to represent.
 22. The system according to claim15, wherein the at least one portion of data is configured to be aversion of a data file within the received data stream; and the at leastanother portion of data is configured to be another version of the datafile within the received data stream; wherein the version of the datafile and the another version of the data file are compressed.
 23. Thesystem according to claim 22, wherein the storing further comprisesstoring compressed versions of the data file in at least one storagelocation.
 24. The system according to claim 23, wherein instructionsfurther comprise uncompressing the representative of the at least oneportion of data and the at least another portion of data; retrieving,based on the uncompressing, at least one of an uncompressed version andan uncompressed another version of the data file from the at least onestorage location.
 25. The system according to claim 24, wherein aprocessing time for the retrieving is based on a number ofrepresentatives configured to represent at least one transaction in theplurality of transactions and configured to be uncompressed to retrievethe uncompressed version.
 26. The system according to claim 15, whereininstructions further comprise repeating the determining, the clustering,and the selecting for each portion of the data in the received datastream.
 27. The system according to claim 15, wherein instructionsfurther comprise storing portions of data that are not selected as arepresentative.
 28. The system according to claim 15, wherein theclustered at least one portion of data and the clustered at leastanother portion of data are configured to have no dependencies on oneanother.
 29. A computer program product comprising machine-readablemedium storing instructions that, when executed by the at least oneprocessor, cause the at least one processor to perform operationscomprising: receiving a data stream, wherein the data stream includes aplurality of transactions, each transaction in the plurality oftransactions includes at least one portion of data from the receiveddata stream; determining whether the at least one portion of data withinat least one transaction in the plurality of transactions issubstantially similar to at least another portion of data within the atleast one transaction; clustering together the at least one portion ofdata and at least another portion of data within the at least onetransaction; selecting one of the at least one portion of data and theat least another portion of data as being a representative of the atleast one portion of data and the at least another portion of data inthe received data stream; and storing each representative of a portionof data from each transaction in the plurality of transactions, whereina plurality of representatives is configured to form a chainrepresenting the received data stream.
 30. The computer program productaccording to claim 29, wherein the at least one portion of data is atleast one compressed portion of data and the at least another portion ofdata is at least another compressed portion of data.
 31. The computerprogram product according to claim 29, wherein the representative is acompressed representative of the at least one portion of data and the atleast another portion of data.
 32. The computer program productaccording to claim 29, wherein the representative is a delta-compressedrepresentative of the at least one portion of data and the at leastanother portion of data.
 33. The computer program product according toclaim 29, wherein the portions of data that are not selectedrepresentatives are configured to be independent of each other, therebyreducing a number of dependencies among the portions of data that arenot selected representatives.
 34. The computer program product accordingto claim 29, wherein instructions further comprise determining whether athird portion of data included in at least one transaction within theplurality of transaction is designated for deletion; determining arepresentative that designates the third portion of data; and deletingthe representative that designates the third portion of data, anddeleting all portions of data that the representative that designatesthe third portion of data is configured to represent.
 35. The computerprogram product according to claim 29, wherein instructions furthercomprise determining whether a third portion of data included in atleast one transaction within the plurality of transaction is designatedfor deletion; determining a representative that designates the thirdportion of data; and deleting the third portion of data without deletingthe representative that designates the third portion of data and otherportions of data that the representative that designates the thirdportion is configured to represent.
 36. The computer program productaccording to claim 29, wherein the at least one portion of data isconfigured to be a version of a data file within the received datastream; and the at least another portion of data is configured to beanother version of the data file within the received data stream;wherein the version of the data file and the another version of the datafile are compressed.
 37. The computer program product according to claim36, wherein the storing further comprises storing compressed versions ofthe data file in at least one storage location.
 38. The computer programproduct according to claim 37, wherein instructions further compriseuncompressing the representative of the at least one portion of data andthe at least another portion of data; retrieving, based on theuncompressing, at least one of an uncompressed version and anuncompressed another version of the data file from the at least onestorage location.
 39. The computer program product according to claim38, wherein a processing time for the retrieving is based on a number ofrepresentatives configured to represent at least one transaction in theplurality of transactions and configured to be uncompressed to retrievethe uncompressed version.
 40. The computer program product according toclaim 29, wherein instructions further comprise repeating thedetermining, the clustering, and the selecting for each portion of thedata in the received data stream.
 41. The computer program productaccording to claim 29, wherein instructions further comprise storingportions of data that are not selected as a representative.
 42. Thecomputer program product according to claim 29, wherein the clustered atleast one portion of data and the clustered at least another portion ofdata are configured to have no dependencies on one another.